**font_properties (new in 3.01)**

A new requirement for training in 3.01 is a font_properties file. The purpose of this file is to provide font style information that will appear in the output when the font is recognized. The font_properties file is a text file specified by the -F filename option to mftraining.

Each line of the font_properties file is formatted as follows:

<fontname> <italic> <bold> <fixed> <serif> <fraktur>
where <fontname> is a string naming the font (no spaces allowed!), and <italic>, <bold>, <fixed>, <serif> and <fraktur> are all simple 0 or 1 flags indicating whether the font has the named property.

When running mftraining, each .tr filename must match an entry in the font_properties file, or mftraining will abort. At some point, possibly before the release of 3.01, this matching requirement is likely to shift to the font name in the .tr file itself. The name of the .tr file may be either fontname.tr or [lang].[fontname].exp[num].tr.

**Example:**

font_properties file:

timesitalic 1 0 0 1 0
shapeclustering -F font_properties -U unicharset eng.timesitalic.exp0.tr
mftraining -F font_properties -U unicharset -O eng.unicharset eng.timesitalic.exp0.tr 
Note that in 3.03, there is a default font_properties file, that covers 3000 fonts (not necessarily accurately) in training/langdata/font_properties.

**Clustering**

When the character features of all the training pages have been extracted, we need to cluster them to create the prototypes.

The character shape features can be clustered using the shapeclustering, mftraining and cntraining programs:

**shapeclustering (new in 3.02)**

shapeclustering should not be used except for the Indic languages.

shapeclustering -F font_properties -U unicharset lang.fontname.exp0.tr lang.fontname.exp1.tr ...
shapeclustering creates a master shape table by shape clustering and writes it to a file named shapetable.

**mftraining**

mftraining -F font_properties -U unicharset -O lang.unicharset lang.fontname.exp0.tr lang.fontname.exp1.tr ...
The -U file is the unicharset generated by unicharset_extractor above, and lang.unicharset is the output unicharset that will be given to combine_tessdata.

mftraining will output two other data files: inttemp (the shape prototypes) and pffmtable (the number of expected features for each character). In versions 3.00/3.01, a third file called Microfeat is also written by this program, but it is not used. Later versions don't produce this file.

NOTE: mftraining will produce a shapetable file if you didn't run shapeclustering. You must include this shapetable in your traineddata file, whether or not shapeclustering was used.

**cntraining**

cntraining lang.fontname.exp0.tr lang.fontname.exp1.tr ...
This will output the normproto data file (the character normalization sensitivity prototypes).

vi /etc/profile

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_77
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

source /etc/profile
java -version

sudo sh
cd /usr/local/
chown YourUserName:staff apache-ant-1.9.6
ln -s apache-ant-1.9.6 ant
vi /etc/profile

export ANT_HOME=/usr/local/ant
export PATH=${PATH}:${ANT_HOME}/bin

source /etc/profile
ant -version

<configuration>
<property>
  <name>http.agent.name</name>
    <value>myNutch</value>
      <description>HTTP 'User-Agent' request header. MUST NOT be empty -
        please set this to a single word uniquely related to your organization.
  NOTE: You should also check other related properties:
    http.robots.agents
        http.agent.description
            http.agent.url
                http.agent.email
                    http.agent.version
  and set their values appropriately.
  </description>
  </property>
</configuration>

ant

ant clean

unzip solr-4.10.4.zip

cp apache-nutch-1.10/runtime/local/conf/schema-solr4.xml solr-4.10.4/example/solr/collection1/conf

rm –f solr-4.10.4/example/solr/collection1/conf/schema.xml

<copyField source="latLon" dest="location"/>

mv solr-4.10.4/example/solr/collection1/conf/ schema-solr4.xml solr-4.10.4/example/solr/collection1/conf/ schema.xml

cd solr-4.10.4/example

输入以上两个命令显示了它们各自的使用方法，后面会详细讲解一部分常用的命令，如下图所示：
![](http://7xi6qz.com1.z0.glb.clouddn.com/ocrsearchcrawl%E5%92%8Cnutch%E5%91%BD%E4%BB%A4%E7%94%A8%E6%B3%95%E8%AF%A6%E8%A7%A3.png)


查看crawl的使用方法：
>-i|index用于告知nutch将抓取的结果添加到配置的索引器中。

>-D用于配置传递给Nutch调用的参数，我们可以将索引器配置到这里。

>Seed Dir种子文件目录，用于存放种子URL，即爬虫初始抓取的URL。

>Crawl Dir抓取数据的存放路径。

>Num Rounds循环抓取次数。

使用示例：

>进入Nutch的runtime/local目录，新建一个urls文件夹：

>在urls文件夹中新建一个存放url的种子文件,seed.txt

>向urls/seed.txt添加初始抓取的URL：http://www.163.com

>开启Solr服务，否则不能正常在Solr中建立索引

>这条命令中，-i告知爬虫将抓取的内容添加到给定的索引中，solr.server.url=http://localhost:8983/solr/是Solr索引器的地址，urls/为种子URL文件路径，TestCrawl为Nutch用于存储抓取数据的文件夹（包含URL、抓取的内容等数据），这里的参数2表示循环抓取两次。

>通过执行上面一条命令，就可以开始抓取网页了。在浏览器中输入http://<host>:8983/solr，选择collection1，就可以在里面通过关键字搜索到已经建立索引的内容。这里需要注意的是，爬虫并没有将指定URL的全部页面抓取下来，查看抓取情况的具体方法请参考下面的分布式抓取。

抓取成功后如下图所示：

![](http://7xi6qz.com1.z0.glb.clouddn.com/ocrsearchnutch%E4%B8%80%E7%AB%99%E5%BC%8F%E6%8A%93%E5%8F%96%E6%88%90%E5%8A%9F.png)

##### 分布式抓取
有的时候，一站式抓取并不能很好的满足我们的需求，因此，这里给大家介绍一下分布式抓取的方法：分布式抓取的实际抓取过程包含多个命令的，为了简化操作，crawl把多个命令组合到一起提供给用户使用的。如果要深入学习Nutch爬虫技术，仅仅会使用crawl命令是不够的，还需要对抓取的过程非常熟悉，这里需要用到上一教程中seed.txt所保存的URL信息，还需删除data/crawldb，data/linkdb和data/segments文件夹下的内容，因为我们要分步重新抓取数据。

###### Nutch数据文件夹组成
执行crawl命令之后，会在Nutch的runtime/local下面生成一个TestCrawl文件夹，里面包含三个文件夹：crawldb、linkdb和segments。

crawldb：它包含Nutch所发现的所有URL，它包含了URL是否被抓取、何时被抓取的信息。

linkdb：它包含了Nutch所发现的crawldb中的URL所对应的全部链接，以及源URL和锚文本。

segments：里面包含多个以时间命名的segment文件夹，每个segment就是一个抓取单元，包含一系列的URL，每个segment又包含如下文件夹：

    crawl_generate：待抓取的URL

    crawl_fetch：每个URL的抓取状态

    content：从每个URL抓取到的原始内容

    parse_text：从每个URL解析得到的文本

    parse_data：从每个URL解析得到的外链和元数据

    crawl_parse：包含外链URL，用来更新crawldb
    
###### 将URL列表注入到crawldb中

###### 生成抓取列表

为了抓取指定URL的页面，我们需要先从数据库（crawldb）里生成一个抓取列表：

generate命令执行之后，会生成一个待抓取页面的列表，抓取列表存放在一个新建的segment路径中。segment的文件夹根据创建的时间进行命名（本教程文件夹名为201507151245）。

generate还有很多可选参数，读者可以通过以下命令自行查看（其它命令的查看方法也一样）：

bin/nutch fetch data/segments/201507151245  #这里的201507151245为文件夹名，需要根据自己的情况进行更改，或者直接采用data/segments文件夹，这样的操作对segments文件夹下的所有子文件夹生效，后文同理。

bin/nutch parse data/segments/201507151245

bin/nutch updated data/crawldb –dir data/segments/201507151245

bin/nutch invertlinks data/linkdb –dir data/segments/201507151245

bin/nutch index data/crawldb -linkdb data/linkdb -params solr.server.url=http://localhost:8983/solr -dir data/segments/201507151245

bin/nutch dedup

bin/nutch clean –D solr.server.url=http://192.168.1.11:8983/solr data/crawldb

Usage: CrawlDbReader <crawldb> (-stats | -dump <out_dir> | -topN <nnnn> <out_dir> [<min>] | -url <url>)
<crawldb>directory name where crawldb is located
-stats [-sort] print overall statistics to System.out
[-sort]list status sorted by host
-dump <out_dir> [-format normal|csv|crawldb]dump the whole db to a text file in <out_dir>
[-format csv]dump in Csv format
[-format normal]dump in standard format (default option)
[-format crawldb]dump as CrawlDB
[-regex <expr>]filter records with expression
[-retry <num>]minimum retry count
[-status <status>]filter records by CrawlDatum status
-url <url>print information on <url> to System.out
-topN <nnnn> <out_dir> [<min>]dump top <nnnn> urls sorted by score to <out_dir>
[<min>]skip records with scores below this value.
This can significantly improve performance.

```

得到的统计结果如下：

TOTAL urls表示URL总数，retry表示重试次数，mins score为最低分数，max score为最高分数，status 1 (db_unfetched)为未抓取的数目，status 2 (db_fetched)为已抓取的数目。

###### readlinkdb


readlinkdb用于导出全部URL和锚文本，查看用法：

这里的dump和url参数与readdb命令同理，导出数据：

将数据导入到linkdb_dump这个文件夹中，查看导出的数据信息：

可以看到，导出的信息类似以下格式：
```http://archive.apache.org/dist/nutch/	Inlinks:
fromUrl: http://www.sanesee.com/article/step-by-step-nutch-introduction anchor: http://archive.apache.org/dist/nutch/

bin/nutch readseg
Usage: SegmentReader (-dump ... | -list ... | -get ...) [general options]
* General options:
-nocontentignore content directory
-nofetchignore crawl_fetch directory
-nogenerateignore crawl_generate directory
-noparseignore crawl_parse directory
-noparsedataignore parse_data directory
-noparsetextignore parse_text directory
* SegmentReader -dump <segment_dir> <output> [general options]
  Dumps content of a <segment_dir> as a text file to <output>.
<segment_dir>name of the segment directory.
<output>name of the (non-existent) output directory.
* SegmentReader -list (<segment_dir1> ... | -dir <segments>) [general options]
  List a synopsis of segments in specified directories, or all segments in
  a directory <segments>, and print it on System.out
<segment_dir1> ...list of segment directories to process
-dir <segments>directory that contains multiple segments
* SegmentReader -get <segment_dir> <keyValue> [general options]
  Get a specified record from a segment, and print it on System.out.
<segment_dir>name of the segment directory.
<keyValue>value of the key (url).
Note: put double-quotes around strings with spaces.

bin/nutch readseg -dump data/segments/20150715124521 segment_dump

cat segment_dump /*

composer require silex/silex twig/twig thiagoalessio/tesseract_ocr:dev-master